Ordinal Rankings on Measures Annihilating Thin Sets

We assign a countable ordinal number to each probability measure which annihilates all H-sets. The descriptive-set theoretic structure of this assignment allows us to show that this class of measures is coanalytic non-Borel. In addition, it allows us to quantify the failure of Rajchman's conjecture. Similar results are obtained for measures annihilating Dirichlet sets. A closet subset E of the unit circle T R/Z is called an H-set if there exists a sequence {nk} of positive integers tending to oo and an interval (i.e., a nonempty open arc) I C T such that for all k and all x E E, nkx 0 I. These sets play a fundamental role as examples of sets of uniqueness for trigonometric series [KL; Z, Chapters IX, XII]. A (Borel) probability measure ,u on T is called a Rajchman measure if ,(n) -+ 0 as Il -+ oo, where ,u(n) = fTTe(-nx)dp(x), e(x) = e2rix. We denote by R the class of such measures. These measures have also been very important o the study of sets of uniqueness. In particular, every Rajchman measure annihilates every set of uniqueness, hence every H-set. After establishing these relationships [Rl, R2], Rajchman conjectured that, in fact, the only measures which annihilate all H-sets are those in R. This, however, is false [Li, L2, L3, L5]. Here, we shall quantify how distant Rajchman's conjecture is from the truth. Given a class ' of closed subsets of T, denote by WJ" the class of probability measures on T which annihilate all sets in i?: , E F" X* VE E ' (,(E) = 0). Thus R C H', where H denotes the class of H-sets. Denote by PROB(T) the compact, metrizable space of (Borel) probability measures on T with the weak* topology. It is easy to check that R is a Borel, in fact II3 (i.e., F6), subset of this space. We establish in ?3 that H' is a rI1 (i.e., coanalytic) but not Borel subset of PROB(T). This is the first example of a natural class of measures of such complexity known to the authors and it highlights the distinction between R and H'. Our method of proof actually provides quite a bit of further information on the relationship between R and H'. In ?1, we assign to each ,u E H' a countable ordinal number h(s) which measures in some sense the complexity of the verification that ,u annihilates all H-sets. We show that h has certain definability properties, namely, it is a Il1-rank (see [KL]). In ?3, using the techniques developed in [L3 and L4], we prove that the rank h is unbounded in w1, the first uncountable ordinal; Received by the editors September 10, 1987. 1980 Mathematics Subject Classification (1985 Revision). Primary 43A46, 03E15; Secondary 43A05, 42A63, 54H05.